Desallocation memory response time

hi to all,
I have encountered some memory desallocation problem.
Is there somebody that knows if Doors is known for having desallocation efficiency problems?

I have to treat a lot of modules, so I open them in read only, treat the data and close the module.
Modules sizes are between 100Mo and 1Go. Do you have an idea about the size needed in RAM to open a 1 Go Modules?
Do yo know the time needed to fully restitute the memory needed in RAM to open a Module

I have the same questions on dynamic allocation, for instance on SkipList.
When delete a SkipList, do you know the time need to restitue the RAM taken by it ?

Maybe is there some commands I ignore to restitute RAM more efficiently?

Thanks for your help

xavier
xavierDxl - Thu Mar 04 05:58:43 EST 2010

Re: Desallocation memory response time
SystemAdmin - Thu Mar 04 08:38:03 EST 2010

Yes, DOORS is a memory hog, especially all strings will be left in memory as there is no garbage collection. Buffers can be deleted from memory, but often the effect is not so large, as you have to handle the data as strings at some point.

What fo you mean by 100 Mo or 1 Go??

As for skip lists: if you insert string values into skip lists, the individual strings will be in memory even though you delete the skip list.

Do not know about timings for these operations, but generally, the more memory you consume, the slower everything works.

Attached is an old Telelogic technical tip paper on DXL memory consumption.
Attachments

attachment_14442058_Guidelines4MemoryMgmtinDXL.pdf

Re: Desallocation memory response time
xavierDxl - Thu Mar 04 08:45:30 EST 2010

SystemAdmin - Thu Mar 04 08:38:03 EST 2010
Yes, DOORS is a memory hog, especially all strings will be left in memory as there is no garbage collection. Buffers can be deleted from memory, but often the effect is not so large, as you have to handle the data as strings at some point.

What fo you mean by 100 Mo or 1 Go??

As for skip lists: if you insert string values into skip lists, the individual strings will be in memory even though you delete the skip list.

Do not know about timings for these operations, but generally, the more memory you consume, the slower everything works.

Attached is an old Telelogic technical tip paper on DXL memory consumption.

thanks for our answer.
"What fo you mean by 100 Mo or 1 Go??" => I mean that the size of the modules I handle vary from 100Mo to 1Go max, depending the module (some of them are about 100Mo, some others 500Mo and some others 1Go)

Thank you very much for your support

regards,
xavier

Re: Desallocation memory response time
llandale - Thu Mar 04 14:05:22 EST 2010

An individual module isn't going cause serious degredation.

One problem we found is that residually opened modules stay open, such as when you follow a link or access dxlAttr or Layouts that do the same thing. I've got a small suite of 'close everything up' that when 'too many' modules are currently open, it closes all modules that were not open when the script began. I arbitrarily set 'too many' to 40.

Another is to be sure to open modules Invisibly; this is especially true if you have remote custom 'addins' defined. When visible, critical also is to bypass the default view and load the standard view. Routinely do this: Module mod = read(NameModFull, false, true).

String parsing is a hog. Avoid concatenation inside a loop. Look at this code. See the completely rediculous explosion in response when concatenating strings, between 10,000 loops and 12,000 loops; from a palty 2 seconds to almost 2 minutes. Whereas the Buffer solution has linear response.

/* Start Mem Use = 900mb
    Results:
NumLoops = 20,000
    109 mSecs, Wasted space =    108890
  xxxx far too long

NumLoops = 12,000
     78 mSecs, Wasted space =     60890
111,922 mSecs, Wasted space = 349215495

NumLoops = 10,000
     62 mSecs, Wasted space =     48890
  2,188 mSecs, Wasted space = 239429495

*/
pragma runLim, 0
int     i
int     NumLoops = 10000
int     Wasted = 0
string  Result = ""
string  Temp
Buffer  bufResult = create()
bool    DoBuff = confirm("Do buffer Test")
// int          TimeStart = intOf(today())
int     TimeStart = getTickCount_()
 
for (i=0; i<NumLoops; i++)
{  if (DoBuff)
   {  Temp = i ""
      bufResult += " "
      bufResult += i ""
      Wasted += length(Temp) +1
   }
   else
   {  Result = Result " " i ""
      Wasted += length(Result)
   }
}
print (getTickCount_() - TimeStart) " mSecs, Wasted space = " Wasted "\n"
 
delete(bufResult)

 


I'm suspecting that DOORS reserves a big string table space when it starts, and slows down drastically when that space is exhausted and more is requested, incrementally, from the OS.

 

 

 

  • Louie

 

 

Re: Desallocation memory response time
SystemAdmin - Mon Mar 12 20:40:31 EDT 2012

llandale - Thu Mar 04 14:05:22 EST 2010

An individual module isn't going cause serious degredation.

One problem we found is that residually opened modules stay open, such as when you follow a link or access dxlAttr or Layouts that do the same thing. I've got a small suite of 'close everything up' that when 'too many' modules are currently open, it closes all modules that were not open when the script began. I arbitrarily set 'too many' to 40.

Another is to be sure to open modules Invisibly; this is especially true if you have remote custom 'addins' defined. When visible, critical also is to bypass the default view and load the standard view. Routinely do this: Module mod = read(NameModFull, false, true).

String parsing is a hog. Avoid concatenation inside a loop. Look at this code. See the completely rediculous explosion in response when concatenating strings, between 10,000 loops and 12,000 loops; from a palty 2 seconds to almost 2 minutes. Whereas the Buffer solution has linear response.

/* Start Mem Use = 900mb
    Results:
NumLoops = 20,000
    109 mSecs, Wasted space =    108890
  xxxx far too long

NumLoops = 12,000
     78 mSecs, Wasted space =     60890
111,922 mSecs, Wasted space = 349215495

NumLoops = 10,000
     62 mSecs, Wasted space =     48890
  2,188 mSecs, Wasted space = 239429495

*/
pragma runLim, 0
int     i
int     NumLoops = 10000
int     Wasted = 0
string  Result = ""
string  Temp
Buffer  bufResult = create()
bool    DoBuff = confirm("Do buffer Test")
// int          TimeStart = intOf(today())
int     TimeStart = getTickCount_()
 
for (i=0; i<NumLoops; i++)
{  if (DoBuff)
   {  Temp = i ""
      bufResult += " "
      bufResult += i ""
      Wasted += length(Temp) +1
   }
   else
   {  Result = Result " " i ""
      Wasted += length(Result)
   }
}
print (getTickCount_() - TimeStart) " mSecs, Wasted space = " Wasted "\n"
 
delete(bufResult)

 


I'm suspecting that DOORS reserves a big string table space when it starts, and slows down drastically when that space is exhausted and more is requested, incrementally, from the OS.

 

 

 

  • Louie

 

 

Louie: Are you able to share the technique you use to get a handle on what modules are currently open so that they can be closed to release memory?

Everybody: I'm running into an out of memory situation with a "for item in current Project do" loop that opens every module and loads every view. There are hundreds of modules and thousands of links to lots of other modules. Even though I close each module after I process it, after about 130 modules I hit the 2GB mark and DOORS dies. I suspect that other modules are being opened (and remain open) due to layout DXL in views and are not being closed. I have been careful to avoid strings and use buffers instead. Thoughts?

Thanks!
Bob3

Re: Desallocation memory response time
SystemAdmin - Tue Mar 13 09:08:00 EDT 2012

SystemAdmin - Mon Mar 12 20:40:31 EDT 2012
Louie: Are you able to share the technique you use to get a handle on what modules are currently open so that they can be closed to release memory?

Everybody: I'm running into an out of memory situation with a "for item in current Project do" loop that opens every module and loads every view. There are hundreds of modules and thousands of links to lots of other modules. Even though I close each module after I process it, after about 130 modules I hit the 2GB mark and DOORS dies. I suspect that other modules are being opened (and remain open) due to layout DXL in views and are not being closed. I have been careful to avoid strings and use buffers instead. Thoughts?

Thanks!
Bob3

for m in database do {
...
}
where:

m is a variable of type Module

Operation
Assigns the variable m to be each successive open module (for which the user has read access) in the database.
or
for m in project do {
...
}
where:

m is a variable of type Module
project is a variable of type Project

Re: Desallocation memory response time
SystemAdmin - Tue Mar 13 13:30:14 EDT 2012

SystemAdmin - Tue Mar 13 09:08:00 EDT 2012
for m in database do {
...
}
where:

m is a variable of type Module

Operation
Assigns the variable m to be each successive open module (for which the user has read access) in the database.
or
for m in project do {
...
}
where:

m is a variable of type Module
project is a variable of type Project

SeanF, thanks for your response.

My concern with this approach is that if there are a lot of modules, this loop could increase my script's runtime. You see, I am already interating through all the modules in the project. With this approach, I would be iterating through all the modules in the project (closing any that are open) as many times as there are modules in the project.

Is there a more nuanced way of only closing those modules that are open (visibly and invisibly), rather than iterating through all modules?

Re: Desallocation memory response time
SystemAdmin - Tue Mar 13 13:53:21 EDT 2012

SystemAdmin - Tue Mar 13 13:30:14 EDT 2012
SeanF, thanks for your response.

My concern with this approach is that if there are a lot of modules, this loop could increase my script's runtime. You see, I am already interating through all the modules in the project. With this approach, I would be iterating through all the modules in the project (closing any that are open) as many times as there are modules in the project.

Is there a more nuanced way of only closing those modules that are open (visibly and invisibly), rather than iterating through all modules?

You could use a call to evalTop_ in startup.dxl to create a top context skip list with addModule and removeModule functions.

You could then set a database wide post open module trigger to call add module and a database wide post close module trigger to call removeModule

You would then always have a skip list of all modules which you could query using funtions such as find rather than a loop but I think this would be unnecessarily complicated so I would suggest just using the database loop.

Re: Desallocation memory response time
llandale - Tue Mar 13 16:15:50 EDT 2012

SystemAdmin - Mon Mar 12 20:40:31 EDT 2012
Louie: Are you able to share the technique you use to get a handle on what modules are currently open so that they can be closed to release memory?

Everybody: I'm running into an out of memory situation with a "for item in current Project do" loop that opens every module and loads every view. There are hundreds of modules and thousands of links to lots of other modules. Even though I close each module after I process it, after about 130 modules I hit the 2GB mark and DOORS dies. I suspect that other modules are being opened (and remain open) due to layout DXL in views and are not being closed. I have been careful to avoid strings and use buffers instead. Thoughts?

Thanks!
Bob3

They've scolded me for posting too many actual deployed functions.
[1] Perhaps it doesn't matter, but before you start get a list of all originally opened modules
  • Skip g_skpOriginal = create()
  • for mod in database do{put(g_skpOriginal, mod, mod)}
[2] "periodically" close the residually open modules
  • for mod in database do
  • { if (find(g_skpOriginal, mod)) continue // don't close one that was already open
  • .. save(mod)
  • .. close(mod) // or maybe close(mod, false) don't save
  • }
The problem becomes how often is "periodically". If you do it every module you will end up doing lots of background opens and closes, since a linked module is probably linked from somewhere else. I think my algorithm arbitrarily counts the open modules, and if there are 40 or more then close the residually open ones. This check is made after each module your script is mindfully opening.

My original algorithm recursively found modules in each folder, and when I switched folders I'd close ther residually open ones. I figured modules in the same folder typically have the same target modules (parent requirements) and the same source linked modules (test procedures).

I made a feeble effort at trying to prioritize which ones to close by remembering which ones have been opened a lot. Thus, commonly used modules would remain open. Didn't seem worth it.

-Louie

Re: Desallocation memory response time
SystemAdmin - Wed Mar 14 05:47:40 EDT 2012

SystemAdmin - Tue Mar 13 13:30:14 EDT 2012
SeanF, thanks for your response.

My concern with this approach is that if there are a lot of modules, this loop could increase my script's runtime. You see, I am already interating through all the modules in the project. With this approach, I would be iterating through all the modules in the project (closing any that are open) as many times as there are modules in the project.

Is there a more nuanced way of only closing those modules that are open (visibly and invisibly), rather than iterating through all modules?

Hi Rob,

The loop

for m in database do {
...
}

only iterates over the open modules, not all the modules, so I am not sure why you think it is not fit for the purpose you describe (identifying open modules so they can be closed). Sorry if I am not understanding something in your request.